cl: fix Gloas checkpoint sync with external execution clients - #22683

@yperbasis I reset this PR to its original three-commit scope so it can receive a focused re-review:

• preserve the skipped checkpoint state root
• validate Gloas payloads with an external EL
• encode empty Engine API transactions as an array

The later downloader, recovery, coordinator, payload-attestation, DA/PeerDAS, persistence, and resolver work—and the review discussion attached to those changes—has moved to stacked draft #23249.

The current diff is now 19 files / 1,045 changed lines instead of 60 files / 10,904 changed lines. The retained safety work is limited to this path: remote and pending retries are budgeted, NewPayload no longer holds the global fork-choice lock, and the authoritative finalized Gloas root survives the production checkpoint restart artifact. The rewritten head was rebuilt on the latest main and passed the affected normal/race suites, repeated lint runs, and make erigon integration.

The long status comment above describes the extracted branch and is superseded for this PR. Please review the new small diff when convenient.

@yperbasis I reset this PR to its original three-commit scope so it can receive a focused re-review:

• preserve the skipped checkpoint state root
• validate Gloas payloads with an external EL
• encode empty Engine API transactions as an array
• prevent payload-attestation validation from copying/replaying state per concurrent gossip message (the devnet OOM fix)

The later downloader, recovery, DA/PeerDAS, persistence/journal, canonical-ingress, and resolver work—and the review discussion attached to those changes—has moved to stacked draft #23249. The payload-attestation OOM cluster remains here because it is required for stable Gloas devnet operation.

The current diff is now 26 files / 1,855 changed lines instead of 60 files / 10,904 changed lines. The retained safety work is limited to this path: remote and pending retries are budgeted, NewPayload no longer holds the global fork-choice lock, the authoritative finalized Gloas root survives restart, and payload-attestation state-context construction is bounded and shared. The rewritten head passed the affected normal/race suites, repeated lint runs, make erigon integration, and both Gloas Kurtosis suites using glamsterdam-devnet-7 images.

The long status comment above describes the extracted branch and is superseded for this PR. Please review the new small diff when convenient.

@yperbasis @AskAlexSharov Review-feedback disposition after the scope reset:

Retained in this PR and addressed

• OnBlock / external NewPayload consistency: pending-envelope application now happens after the block bookkeeping commit; the fork-choice mutex is released only around the EL call, state/block identity is refreshed after relock, duplicate completion is rechecked, failed work is retained and retried, and block-event optimism is emitted after the result is known. The reverse/concurrent and DB-index retry paths have focused tests.
• External EL retry monopolization: pending work is capped at 32 attempts under one aggregate 2 s context, failures rotate fairly across both pending origins, cancellation and DB failures are retained, and no network call holds f.mu. The synchronous bound is intentional so completion is visible before later recovery/verification phases; it cannot expand to one 30 s timeout per queued item.
• Forward validation for local versus external EL: validation is intentionally based on the presence of an execution client, while SupportInsertion() remains the separate insertion/flush capability. Insertion alone does not establish the NewPayload validation result needed by fork choice.
• Skipped/legacy checkpoint identity: the fork-graph constructor establishes the authoritative previous state root before dumping. The production finalized checkpoint now persists a state-digest-bound root record durably; unsafe legacy same-slot Gloas local resumes fall back instead of inventing an identity. Zero-cache skipped-slot and production restart paths are covered.
• Payload-attestation OOM: same-message service validation is serialized; state-context construction is singleflight per block root, globally limited to one full-state build, and cached as compact data with a 128-entry bound. Cancellation, invalid-to-valid competition, panic cleanup, cache publication, and vote races are covered.
• PeerDAS topic registration: PeerDAS construction no longer subscribes before gossip topics exist. An idempotent Start(ctx) is invoked after RegisterGossipServices, with a regression test proving subscriptions begin only at that boundary.
• Empty Engine API transactions: the required JSON field is always encoded as an array, including an empty one.

Removed from this PR, not dismissed

The backward/forward downloader rewrite, lookahead trust and EMPTY/FULL inference, HTTP/P2P recovery budgets, skipped-envelope persistence, live envelope resolver, envelope journal/prune/startup recovery, canonical-ingress hardening, and observer aggregate behavior are no longer in #22683. They are preserved in stacked draft #23249, which is intentionally not ready to merge. The related High/Medium feedback remains follow-up work there rather than being claimed as fixed by this PR.

This includes the inline findings about local failures banning peers, nil-engine stage blocking, repeated full-state copies, prune error/scan bounds, envelope-before-block handling, Gloas EMPTY skipping, stale lookahead trust, one-block forward progress, resolver hot-path copying, startup scratch artifacts, repeated SSZ hashing, and observer aggregate cost. I replied to each thread with its exact disposition and resolved it only because the referenced code is absent from the current diff.

Low/nit disposition

• Duplicate block reads and envelope validation, dead downloader branches/helpers, response-body lifecycle, retry-sleep duplication, batch rehashing, and metric-placement nits belong to the extracted downloader/journal/coordinator work and are absent from this diff.
• The lazy payload-validation coordinator path is also absent; the slim implementation uses explicit initialized admission state.
• These were not ignored as “unnecessary”; they were excluded because their production code was excluded. They remain review input for the follow-up PRs if those clusters are reintroduced.

Current core diff: 31 files, +1756/-203 (1,959 changed lines). The exact head passed the PeerDAS regression and affected packages, lint twice, and make erigon integration; the previous core head passed both devnet-7 Gloas Kurtosis jobs, which will rerun for this update.

Scope correction: the small PeerDAS startup-order fix is now restored in #22683 as commit fc350dbed1.

The devnet warning was reproducible: NewPeerDas subscribed before RegisterGossipServices created the data-column topic handles, producing topic not found and potentially missing the initial archive/custody subscriptions. PeerDAS now has an idempotent Start(ctx) called after gossip registration; the focused regression and affected packages pass.

The core diff remains close to the requested size at 31 files / 1,959 changed lines. The larger DA retry/resolver lifecycle remains excluded: block state not found is already retained by the core pending-envelope path, while missing FULL envelopes are retried under the bounded chain-tip pass and actively refetched by root. No broader resolver/journal/downloader cluster was restored with this change.

Review follow-up for 969925928d (cl: harden Gloas payload recovery and validation).

I went through the latest feedback and kept the changes within the Gloas payload validation/recovery scope. The actionable findings are addressed as follows:

• Run the complete CL envelope transition validation before engine_newPayload, so a CL-invalid reveal cannot trigger EL work or invalidate the beacon header. Local self-build ingress now also rejects a nil payload before lookup, queueing, or EL access.
• Make execution-payload index persistence idempotent and retry-safe. Same-root writes are coalesced, retries use the authoritative envelope persisted by forkchoice, root-only repair survives temporary disk-read failures, and cancellation/panic/transient storage paths do not report false success or discard repair ownership. Pending OnBlock writes use the same coordinator.
• Bound payload-attestation validation work by protocol cardinality while preserving same-key coalescing and a retry candidate. Context construction is bounded across distinct roots, with cancellation and panic cleanup covered.
• Recover and verify envelopes from the selected fork-choice head rather than the highest-seen side branch. The scans now use bounded cursors, retain partial/failed work, reset on reorgs, preserve progress across direct extensions, process the immediate frontier first, and deduplicate it within the 32-item cycle limit.
• Rotate the first retry class across cycles so pending-envelope, pending-EL, and unverified-anchor work each periodically receive the full remaining two-second budget without increasing concurrency or the total cycle budget.
• Restrict the finalized-state root sidecar behavior to Gloas and later, preserving the legacy pre-Gloas checkpoint path. Anchor restoration also handles legacy non-zero header roots without hashing the already-populated root field.

Validation completed:

• make lint twice: clean (0 issues)
• make erigon integration: passed
• focused forkchoice, stages, network/services, checkpoint-sync, and fork-graph tests: passed
• race-focused tests for the new singleflight, cancellation, retry, attestation admission, and cursor paths: passed
• adversarial Standards/Spec, lifecycle/progress, and boundary reviews converged with no remaining in-scope actionable findings

I intentionally did not fold unrelated pre-existing behavior or broad refactors into this update.

Addressed the two latest review findings in c005feb.

Selected-head envelope recovery:

• A missing selected Gloas head now gets one root-specific peer request during the existing 2s head wait.
• The request is gated by the selected block actual version, so pre-Gloas heads are unchanged.
• The ancestor recovery cursor remains independent, so a legitimate EMPTY head does not pin ancestor progress.
• The stage no longer joins a late requester after the deadline, and a response arriving after cancellation is not applied.

Payload-attestation first-valid handling:

• Removed the pre-signature leader/one-waiter slot that allowed forged candidates to displace a later valid message.
• Full validation admission now applies cancellation-aware backpressure instead of immediately dropping the next candidate.
• After PTC membership and BLS verification, fork choice atomically commits the first valid (slot, validator) and its votes under the vote lock, including candidates for different block roots.
• The seen state resets forward by slot, rejects stale completions, and is bounded by the 512-member PTC. No candidate pool was added.

Deliberate scope boundary: hardening an internal requester that permanently violates context cancellation would require managed worker ownership, and strict cross-validator scheduling fairness would require pubsub/per-peer scheduler changes. The production requester honors context; validation work is bounded by the existing 512 admission plus libp2p validation queue/throttle. Those broader redesigns are not included here.

Validation:

• go test ./cl/phase1/forkchoice ./cl/phase1/network/services ./cl/phase1/stages -count=1
• focused go test -race across the new concurrency/recovery cases
• make lint, repeated clean
• make erigon integration
• three independent Standards/spec, adversarial, and lifecycle review passes converged with no remaining in-scope findings.

Pushed a49b7c84c4 to this PR branch.

What changed:

• Merged the latest main into the PR branch.
• Fixed the merge-tree CI failure by migrating the removed db/kv/memdb test helper to db/kv/mdbx/mdbxtest.
• Preserved the PR's execution-payload cancellation/error semantics while resolving the merge conflicts.
• Made selected-head envelope requests retryable after error/empty completion, with per-root attempt tokens so stale A→B→A completions cannot release a newer claim.
• Made bounded ancestor recovery advance past unavailable holes and revisit them on later full passes, so one unavailable envelope cannot permanently block deeper recovery.

TDD / validation:

• Reproduced the original CI error before the test-helper migration, then confirmed the focused test turned green.
• Added regression coverage for selected-head retry/release behavior and recovery cursor progress.
• go test ./cl/phase1/forkchoice ./cl/phase1/network/services ./cl/phase1/stages -count=1
• Focused stage race tests passed.
• make erigon integration passed.
• make lint reports no PR-related issues; the only local macOS diagnostic is the current-main Linux-only db/seg.residencyOnce field, which is used by the Linux implementation and is unrelated to this PR.
• No test skips were added.

Two independent adversarial/lifecycle reviews converged on the changed recovery surface. GitHub checks are restarting from the new head.